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ABSTRACT 



This study tested the amount of value added to critical and 
communication skills by the university experience using a strategy that 
compared the skills of entering and graduating students, ^York University 
(Ontario). The study involved, first, identifying skills that might be- 
improved over the course of a university education; second, developing survey 
questions that measured skills for entering and graduating students at four 
faculties; and third, performing covariance analysis of survey results for 
entering and graduating students. Data were generated by three 
questionnaires, with response rates ranging from 55 to 58 percent for two 
surveys in the fall of 1995 to 58 percent one conducted in the summer of 
1996. Eight tables detail skill categories and topics; list characteristics 
of survey respondents; correlate skills and grades for entering students, for 
graduating students, by gender, by ethnic origin, and by home language; and 
provide Z-scores to assess value added for entering and graduating students. 
Overall, graduating students were found to have better-developed skills than 
entering students. The paper also focuses on the rationale for the relatively 
cost-effective research design. (Contains 30 references.) (CH) 
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Institute for Social Research 



Founded in 1965, the Institute for Social Research (ISR) is an Organized Research Unit of York 
University. The Institute's purpose is to promote, undertake and critically evaluate applied social 
research. The Institute provides consultation and support services to York faculty, students and staff 
conducting research in the social sciences, and, to a lesser extent, in the biological and physical sciences. 
For researchers from other universities, government agencies, public organizations and the private sector, 
the Institute provides consultation on research design and undertakes data collection, data processing and 
statistical analysis, on a fee-for-service basis. 

ISR houses the largest university-based survey research unit in Canada, annually conducting 
twenty to thirty research projects ranging from small surveys in one locale to provincial and national 
surveys. The capabilities of the Institute include questionnaire and sample design, sample selection, data 
collection, preparation of machine-readable data files, statistical analysis and report writing. 

ISR's Statistical Consulting Service provides consultation on research design and statistical 
analysis. The Service also sponsors short courses on statistical analysis, research methodology and the 
use of statistical software. The consulting service is partially supported by a grant from the Natural 
Science and Engineering Research Council (NSERC). 

ISR's data archive provides public access to survey data collected by the Institute, to data sets 
from major Canadian surveys, and to official statistics, such as the census aggregate and public-use 
microdata files from the Canadian Census. 



For more information, write to: 

Institute for Social Research 
York University 
4700 Keele Street 

North York, Ontario, Canada M3J 1P3 

Telephone: (416) 736-5061; Fax (416) 736-5749 
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Foreword 

The Institute for Social Research produces four types of articles in its publication series: 

• Working papers; 

• Reports on various technical and managerial aspects of the research process designed for 
technical support staff and research managers; 

• Reports on topics of general interest to non-specialist readers; and, 

• Reports on various methodological and substantive issues aimed at experts in the field. 
The following is a working paper. Comments are welcome. 
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Summary 

Some universities are interested in knowing the value added to skills - e.g. critical and 
communication skills - by the university experience. The best way of obtaining 
information on this matter is through longitudinal analyzes with appropriate control 
groups that utilize both subjective and objective measures of skills; unfortunately, such 
designs are costly and studies based on them take a long time to complete. In this article 
an alternate strategy that involved comparing the skills of entering and graduating 
students at York University is described. It is argued that information collected in this 
fashion can be of assistance in assessing the value added to specific skills and can be 
used in a diagnostic fashion by faculties concerned with skills development. 
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Introduction 



An increasing number of Canadian universities are carrying out surveys of their 
graduates that among other things ask students to comment on development in various 
skills, such as analytic and communication skills, over the course of their education. To 
date, however, few if any attempts have been made to measure the ‘value added’ to skills 
by the university experience. There are several reasons for this. First, there is no 
consensus regarding skills that should be developed over the course of a student’s 
university career. Second, even if agreement were reached on which skills should be 
developed, there is no agreement on ways to measure them. Third, the longitudinal 
studies required to measure value added in skills are expensive and take a long time to 
complete. 

At York University, in order to measure the value added in skills over the course of an 
undergraduate education, steps were taken to deal with each of the three problems. First, 
an iterative process was established in which members of the university community 
identified skills that, ideally, would be improved over the course of a university 
education. Second, survey questions were developed to measure such skills on the part 
of students entering, and graduating from, four faculties. Third, a research design was 
used in which skills of entering students were compared to those of graduating students 
after adjustments had been made through an analysis of covariance for Ontario Academic 
Credit marks, gender, ethno-racial origin, and language spoken in the home while 
growing up. This article focuses on the rationale for conducting an analysis of the value 
added to skills in this relatively cost effective fashion and the results of the research. 



Background to Research 

In a review of exit surveys used in Canadian universities, Evers and OHara (1996:44) 
define knowledge as, “The understanding of a body of information in a particular field; 
for example, electrical engineering.” Skills are seen as, “The abilities or proficiencies 
developed in certain areas; for example, written communication.” Finally, values are 
defined as, “The commonly held positive attitudes toward an abstract concept; for 
example, respect for diversity.” 

Evers and O’Hara (page 54) note that while a growing number of Canadian universities 
use self-reports in exit surveys to measure their graduated knowledge, skills, and values 
(KSV), “there is little evidence of the development of more sophisticated methodologies 
such as behavioral measures or objective testing of KSV.” In addition to devising such 
measures, like many other researchers, particularly in the United States, they conclude 
that it is necessary to develop indicators of the ‘value added’ by the institution to student^ 
knowledge, skills, and values. This objective can only be achieved if information is 
collected on both entering and graduating students. 

There are a number of steps that must be taken if these desiderata are to be achieved. 
First, to focus only on skills, it is necessary to develop agreement regarding the types of 
skills that should be fostered during an undergraduate career. As attempts to achieve this 
objective at the national level have been less than successful in the United States (Pike, 
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1995a; Jones, 1994), it is highly unlikely that we will have any more luck north of the 
border. As a result, it makes sense for individual universities to decide for themselves 
the skills with which they are most concerned. If this can be done in conjunction with 
other universities, so much the better. 

Once individual institutions reach some form of agreement on which skills should be 
developed, the second step is to find ways to measure them. In the cross-sectional exit 
surveys examined by Evers and O'Hara, students are typically asked to self-report on 
how much university experiences have increased skills, such as communication or 
interpersonal skills. 

As well as using self-reports to measure skills, standardized tests, such as the California 
Critical Thinking Skills Test, the Watson-Glaser Critical Thinking Appraisal, the Ennis 
Weir Thinking Essay Test, and the Cornell Critical Thinking Test, can be used to 
measure skills; however, in many instances users must pay to utilize instruments such as 
these. Moreover, incentives must frequently be given to students to complete lengthy test 
modules. For example, Pascarella and Associates (1995) report that in their longitudinal 
study of university outcomes in the United States they started by paying students $25 
U.S. to complete tests and questionnaires and that the stipend increased by $5 each time 
additional tests were taken. Even if standardized tests are used, questions can be raised 
regarding their validity, particularly for measuring short-term gains in skills (McMillan, 
1987; Jacobs, 1995). 

Independent of whether or not self-reports in surveys or standardized tests are used to 
measure skills, when examining how much student^ experiences in any given university 
contribute to the development of skills or any other desired outcome, it is important to 
obtain measures of the outcome under consideration at entry and again at graduation so 
that the value added can be assessed (Astin, 1991). Similar measures should be taken of 
a same-age control group that does not attend university. Increases in skills between 
entering and graduating students that are not also observed in the control group can be 
attributed to the university experience. 

In longitudinal research such as this, because many students might leave the university or 
refuse to participate in both the entry and exit measurement of skills, large samples are 
required to ensure that at graduation sufficient numbers are still involved in the study to 
facilitate meaningful analysis. Even if a large sample remains at graduation the 
possibility exists that non-participants are different from students who remain in the 
study. In addition, control group attrition is a potential problem. Also, when relying on 
longitudinal studies, institutions must wait until a cohort has made its way through the 
university system before a measure of value added is available. 



A Cost Effective Alternative 

Clearly, identifying and measuring skills, and determining the value added by the 
university experience to skills in a longitudinal study is a complicated, potentially costly, 
and time consuming process. There is, however, a way to decrease the cost and amount 
of time needed to obtain information on the value added component of the university 
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experience. Subject to qualifications regarding question wording to be discussed later, 
this approach involves using surveys in which entering and graduating students provide 
self-assessments of skills in particular areas. Through an analysis of covariance, after 
controlling for variables having the potential to influence skills acquisition, such as 
previous levels of achievement (high school marks), gender, language spoken in the 
home, and ethno-racial origin, comparisons between the groups can then be made on 
various measures of skills. If comparisons show that graduating students have skills not 
evident among entering students and a same-age control group, differences between the 
entering and graduating students can be attributed to the university experience. 

Variations of this method have been used successfully by Keeley, Browne and Kreutzer 
(1982) and Steele (1986). 

One problem with this approach is that when they entered university, the graduating 
students may have been different from the entering group with whom they are being 
compared. For example, if in recent years an increased emphasis had been placed on 
certain skills in high schools, entering students would demonstrate higher levels of skills 
than would have been displayed by graduating students when they were in first year. A 
second problem is that students with certain characteristics, low motivation for example, 
may leave the university before completing their studies. As a result, graduating students 
may be different from those entering the university in ways that are hard to detect and 
control. Despite these potential difficulties, Pascarella and Terenzini (1991 : 124) point 
out that in studies carried out on reflective judgement, longitudinal and cross-sectional 
research produce similar results. A similar conclusion was reached by Pace (1979). As a 
result, provided that adjustments are made of possible confounding influences, cross- 
sectional studies involving entering and graduating students can be used by institutions in 
assessing the valued added in particular skills. 

Although both longitudinal and cross-sectional studies require an external control group 
if the possible effects of the university experience are to be disentangled from those of 
maturation, external control groups are most often absent from studies of university 
outcomes. While this presents difficulties if the intent is to assess the effect of the 
university experience per se, provided same-age groups are under consideration, it is less 
problematic if the research objective is one of assessing the impact of different 
institutional contexts on outcomes. For example, if the outcomes of one university, 
faculty, or department are being compared with those of other universities, faculties, or 
departments, provided that same-age groups are involved in the study and that pre-entry 
characteristics are held constant, an external control group is not required. 



Self-Reports 

Self-reports are an integral part of the relatively more cost and time effective response to 
measuring valued added in generic skills currently under discussion; however, some may 
view the use of techniques such as this as less desirable than more ‘objective’ measures of 
skills. As Pike (1995a:l) points out, however, in the United States, the National 
Education Goals Panel Resources Group on Adult Literacy and Lifelong Learning noted 
that the development of assessment tools of use at the national level to measure 
university outcomes would cost several million dollars and take many years to complete. 
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In the interim the Group recommended that self-reports be used as proxies. 

To what extent can self-reports be viewed as reasonable proxies of knowledge and/or 
skills, and/or values? After examining a number of studies in which the results of self- 
reports were compared to test results, Pascarella and Terenzini (1991:100) conclude that 
correlations between self-reports and other measures of the same phenomena range from 
.25 to .65. More importantly, the literature they draw upon in their examination indicates 
that the validity of self-reports varies with the specific knowledge and/or skills, and/or 
values under discussion. For example, Berdie (1971) found a high correlation between 
self-professed knowledge of public figures and the results of tests designed to measure 
knowledge of the same figures; however, the relationships between self-reported and 
tested knowledge of authors and artists were not as high. Similarly, Pohlmann and Beggs 
(1974) discovered that self-reports of academic growth in the affective realm were 
supported by test results. Growth in simple and complex cognitive realms, however, did 
not correlate highly with self-reports. In a review of the literature on the utility of self- 
reports Baird (1976) cites examples of both high and low relationships with external 
measures of various phenomena. Still others have presented information suggesting that 
while self-reports have some uses, they should not be viewed as substitutes for other 
measures of various college or university outcomes (McMorris and Ambrosino, 1973; 
Dumont and Troelstrup, 1980). Overall, despite their limitations and inconsistencies in 
the research, Kuh and Associates (1997:48) argue that self-reports are valid when: 
respondents know the information requested; questions are clear and unambiguous; and 
respondents treat questions seriously. 

In a study conducted by Pike (1995a) conclusions were reached that support the validity 
of the second condition of Kuh and Associates. Briefly, Pike studied 1,568 graduating 
students from 10 colleges and universities in the United States who completed the 
College Basic Academic Subjects Examination (College BASE) adapted for graduating 
students. The College BASE tests proficiency in English, mathematics, science, and 
social science. Students who completed the examination also participated in a survey in 
which they rated their ability on exactly the same matters covered in the examination. 

The correlations between test performance and self-reports were sufficiently high for 
Pike to conclude that self-reports can be used as general proxies for traditional measures 
of academic achievement. Nonetheless, in an earlier publication Pike (1995b) noted that 
self-reports were good proxies for proficiency in mathematics, 'reasonably weak* proxies 
for English and science competency, and very weak indicators of proficiency in social 
studies. 

As a general rule Pike emphasizes that if self-reports are to be valid measures of, in his 
case, knowledge measured in standardized tests, there must be a high content 
correspondence between the self-report questions and those asked in tests . (While this 
may seem self-evident, other research has focussed on the relationship between some 
general self-report measures and the results of specific tests.) If this rule is not followed, 
it may be difficult to specify exactly what self-reports measure. 

Although this and other work by Pike (1994, 1995b) is encouraging, it should be stressed 
that the focus of his research has been on knowledge (not skills) as measured through the 
College BASE. Although research to be analysed in this report assumes a similar 
relationship between self-reported skills and the results of tests designed to measure such 
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skills, research is necessary to confirm the connection. 



An encouraging step in this direction has been taken by Evers and Associates (1993) in a 
study in which they compared the self-reported skill levels of recent graduates of a 
number of Canadian universities to assessments of the same skills by their employers. 
The average correlation for statistically significant relationships between self-reports and 
employers’ assessments was .16. The correlation was highest for written communication 
skills, .26, and lowest for decision making skills, .07. 

An examination of data derived from The International Adult Literacy Survey (ILS) 
(Anonymous, 1995) jointly undertaken by Statistics Canada and The Educational Testing 
Service located in the United States, also sheds light on the relationship between self- 
report and other measures of skills. As well as obtaining self-assessments of reading, 
writing, and quantitative skills, this study required participants to complete a number of 
performance tasks of varying levels of difficulty related to prose literacy, document 
literacy, and quantitative literacy. For Canadians aged 16 to 25 who had completed 
post-secondary education (the group most comparable to the subjects of the current 
study) the correlations between self-assessed reading and test measured prose and 
document literacy were .45 and .36 respectively. The correlations between self-reported 
writing skill and test measured prose and document literacy were .29 and .22; however, 
the last mentioned was not statistically significant. Finally, there was a correlation of .29 
between self-assessed quantitative skills and quantitative literacy as measured in tests. 

The study by Evers and Associates and the results of the ILS survey indicate on the basis 
of Canadian data that there do appear to be statistically significant correlations between 
self-assessments of skills and skills measured in other ways. While the magnitude of 
these correlations is insufficient for making academic decisions about, for example, 
individual university graduates who may be presumed to have mastered this or that skill, 
they are sufficient for analyses of differences in skill development among graduates of 
different departments or universities. 



Because there is no consensus in Canada or the United States regarding the generic skills 
that are both important for university graduates to possess and that should be developed 
throughout a student’s university career, in this study, a list of desirable skills was 
developed through an inductive process that could be utilized easily in other universities. 
In an iterative process, faculty members with knowledge of skills development, along 
with researchers from the Institute for Social Research at York University, identified a 
number of tasks, the performance of which ideally would be improved over the course of 
a university career. 1 Next, questions were developed that focussed on the difficulty 
students would have in completing specific tasks. Finally, questions were grouped into 
logical categories. Although in the ILS study reasonable correlations were found 



'Ron Sheese, Nick Elson, Darla Rhyne, Tammy Chi, David Northrup, and Paul Grayson 
participated in this phase of the study. 



The Important Skills 
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Table 1: Skill Categories and Question Topics 



• Analytic Skills 

• identifying the main points in lectures in your major 

• clearly identifying the pros and cons of controversial issues like abortion 

• figuring out the main arguments in articles written on topics in your discipline 

• identifying flaws in positions given by other students in classes or seminars 

• explaining your strengths and weaknesses to a potential employer 

• defending a position you have taken in a classroom or seminar against the criticisms of other 
students 

• Communication Skills 

• taking an article you read for a course this year and summarizing it in no more than two pages 

• verbally presenting your ideas on a topic of your choice to a group of ten strangers 

• verbally communicating to other students in your classes the flaws in their positions or arguments 

• writing a letter to a friend 

• writing a letter of application for a job 

• expressing yourself clearly in written English in an essay 

• correcting the grammar and spelling in the essay of a friend 

• Personal Skills 

• assessing the feelings of people you have worked with in part-time or summer jobs 

• 'cooling out' a friend who is annoyed with you 

• being able to apologize to someone if you said something wrong about them 

• being able to admit to yourself when you are wrong 

• being able to get along with other people 

• enjoying meeting new people 

• helping friends patch up disagreements 

• considering the feelings of others before doing things 

• knowing yourself 

• Organizational Skills 

• planning a job search strategy for a friend 

• organizing priorities to prevent rushing at the last minute 

• recruiting and organizing twenty strangers to collect money for the Heart and Stroke Fund 

• Comparative Skills 

• comparing what is going on in Canada to what is going on in a third world country 

• comparing what is going on in Canada today to what was going on fifty years ago 

• comparing what is going on in Canada to what is going on in any European country 

• Basic Numeracy Skills 

• determining change from a $10 bill for a $2.75 pen 

• calculating 15% discount on a $9.36 book 

• determining percentage Liberal vote when 15 students vote Conservative, 10 NDP, and 20 Liberal 

• solving for 'x' in the equation 3x-5=56.7 

• explaining the meaning of 'square root' 

• Basic Computer Skills 

• using a word processing program 

• using a spread sheet program 

• using a statistical analysis program 
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between some general questions on skills and test measured skills, in keeping with Pikds 
finding noted earlier, tasks , and their related questions, were very specific and relevant 
to students* experiences. 

Survey questions were tested in a small pilot survey involving students about to graduate 
from Pure and Applied Science, Arts, Fine Arts, and Administrative Studies. For each 
question, on a five point scale, responses of 1 indicated a high, and 5 a low, level of 
difficulty in completing the specific task referenced in the question. On the basis of the 
pilot, some questions were dropped and assessments were made of the reliability of 
indices comprised of various questions. The relevant skill categories and question topics 
used in the construction of indices are displayed in Table 1. 



The Sample 

Information for the analysis of value added in generic skills between first year and 
graduation was collected in two surveys conducted in the Fall of 1995 and one in the 
Spring of 1996. In one of the Fall 1995 surveys, questionnaires assessing skills were 
mailed to all students who had just entered the faculties of Fine Arts, Science, and 
Administrative Studies and to a sample of students entering the Faculty of Arts. The 
numbers of returns were 171, 223, 100, and 812 respectively. The response rate was 
55%. 

The second survey carried out in the Fall 1995 focussed on all students eligible to 
graduate in the Fall convocation. In total, 480 Arts students returned completed 
questionnaires as did 25, 30, and 6 students from the Faculties of Fine Arts, Science, and 
Administrative Studies respectively. Low numbers reflect the facts that compared to Arts 
other faculties are small and relatively few students graduate in the Fall. The response 
rate for the survey was 58%. 

The survey carried out in the Spring and early summer of 1996 included all students from 
the same faculties eligible to graduate in the June convocation. In total, 1,835 Arti 
graduates, 180 graduates from Fine Arts, 165 from Science, and 70 graduates from the 
Schulich School of Business completed the survey for a response rate of 51%. Overall, 
the three surveys included 1,306 entering and 2,791 graduating students. The results of 
all three surveys were merged together and with information, such as grades, obtained 
from administrative records. 

The characteristics of survey respondents are summarized in Table 2. Consistent with 
the sampling procedures outlined above, for both the entering and graduating surveys, the 
vast majority of students were enrolled in the Faculty of Arts. For both surveys nearly 
two thirds of respondents were female. Administrative records indicate that this gender 
distribution is comparable to that for undergraduate students as a whole. With regard to 
ethno-racial origin, the majority of both entering and graduating students identify 
themselves as of European origin; however, there are more students of European origin 
among graduating than entering students. Finally, most students in both surveys spoke 
English in the home while growing up. Nonetheless, considerably more graduates than 
entrants reported English as their home language. 
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Table 2: Characteristics of Survey Respondents 











Group 

Total 


Entrants 


Graduates 


Faculty 


Business 


7.7% 


3.1% 


4.8% 






104 


70 


174 




Arts 


61.7% 


81.6% 


74.1% 






839 


1835 


2674 




FineArts 


13.3% 


8.0% 


10.0% 






181 


180 


361 




Science 


17.3% 


7.3% 


11.1% 






235 


165 


400 


Group Total 


Col % 


100.0% 


100.0% 


100.0% 




Count 


1359 


2250 


3609 


Gender 


Female 


61.0% 


68.0% 


65.4% 






831 


1531 


2362 




Male 


39.0% 


32.0% 


34.6% 






531 


719 


1250 


Group Total 


Col % 


100.0% 


100.0% 


100.0% 




Count 


1362 


2250 


3612 


Ethno-Racial 


Black 


5.3% 


6.1% 


5.8% 


Origin 




72 


135 


207 




South Asian 


8.0% 


4.2% 


5.6% 






108 


94 


202 




Chinese 


16.6% 


5.7% 


9.9% 






225 


128 


353 




Other 


14.8% 


10.6% 


12.1% 






200 


235 


435 




European 


55.3% 


73.4% 


66.6% 






749 


1635 


2384 


Group Total 


Col % 


100.0% 


100.0% 


100.0% 




Count 


1354 


2227 


3581 


Home 


English 


63.9% 


76.4% 


71.7% 


Language 




868 


1722 


2590 




Chinese 


12.8% 


3.0% 


6.7% 






174 


67 


241 




Italian 


3.5% 


6.4% 


5.3% 






48 


144 


192 




Other 


19.7% 


14.2% 


16.3% 






268 


320 


588 


Group Total 


Col % 


100.0% 


100.0% 


100.0% 




Count 


1358 


2253 


3611 
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Control Groups 

While it is not possible to have an external control group for the current study, a means 
of estimating the extent to which maturation affected the skills under examination was 
devised. This objective was accomplished by using the responses to 6,974 questionnaires 
completed for departmental reviews in the Winter and Spring of 1996 and 1997 
(response rate approximately 65%). Among other things, at the departmental level, such 
surveys measure the same skills as in the survey of entering and graduating students. 

In brief, the logic of the procedure for estimating the effects of maturation is based on the 
fact that in all faculties but one at York University, most students enter first year from 
high school. The exception is Atkinson College, the evening operation at York, that 
caters to ‘mature’ students, most of whom are older than ‘traditional’ students in other 
undergraduate faculties. While the vast majority of students entering Atkinson have 
some prior post-secondary education, a minority have, at best, completed high school. 

As a result, differences in skill levels between students entering Atkinson with no more 
than high school education, and traditional entrants to other faculties, can be attributed to 
the maturation process. 

Unfortunately, as with most of the research focussing on value added by the university 
experience, resources did not permit the establishment of an external control group. 

Even if resources had not been an issue, given how some of the specific skill 
constellations under discussion were operationalized, it is doubtful that a comparison 
with an external control group would have been meaningful. For example, being able to 
identify the main points in lectures in your major is irrelevant to someone with no post- 
secondary experience. Moreover, as the objective is to measure the value added in skills 
among graduates of various faculties, a control group is not necessary. 



Reliability of Indices 

The reliability of the various indices of skills as outlined in Table 1 was tested in the 
combined surveys of graduating and entering students by using Cronbach’s alpha which 
“can be viewed as the correlation between this test or scale and all other possible tests or 
scales containing the same number of items, which could be constructed from a 
hypothetical universe of items that measure the characteristic of interesf (Norusis, 
1992:149). In general, indices having alphas at or above .7 can be regarded as reliable. 

With one exception, alphas were well above .7. The coefficient for analytical skills was 
.84; for communication skills .80; personal skills had a coefficient of .81; for 
comparative skills the alpha was .85; the coefficient for basic numeracy skills was .79; 
and for basic computer skills alpha was .78. Only organizing skills, with a coefficient of 
.63, must be treated with caution. 2 



2 For November 1995 graduates of the Faculty of Arts, a principal components analysis 
using all of the skills variables was conducted in which the number of factors specified for 
retention, 7, was equal to the number of indices that had been constructed. With factor loadings 
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Correlations Among Skills and Grades 

Correlations among each of the skills and Ontario Academic Credit (OAC) marks are 
detailed in Tables 3 and 4. The first thing of note is that for entering students (Table 3) 
correlations between OAC marks and various skills are low. Of those that are 
statistically significant, there is a positive correlation between, on the one hand, OACs 
and, on the other, communication skills (.102), organizing skills (.076), numeracy skills 
(.101), and basic computer skills (.065); however, for personal skills the correlation is 
negative ( -.1 14). 

If first year GPA is examined, it is evident that the highest statistically significant 
correlation is with OAC marks (.5 1 9). Statistically significant correlations of GPA with 
skills are weak. Analytic, communication, and organizing skills have correlations of 
.063, .087, and .059 with first year GPA. Personal skills correlate negatively (-.091) with 
GPA. 

There are two ways in which these general finding can be interpreted. First, if one 
expects that generic skills examined here are a necessary condition for high school 
achievement as measured in OAC marks, the low correlations can be taken as an 
indication that the current measures of generic skills are invalid: students with high OAC 
marks must possess high generic skills. On the other hand, if it can be accepted that 
OAC marks do not necessarily reflect generic skills the findings make sense. Certainly 
many faculty who bemoan the lack of preparation on the part of first year students 
despite high OAC marks would be amenable to this interpretation. At the same time, if 
employers are to be believed, it is equally likely that there is little relationship between 
high university grades and skills (Jones, 1994). 

Unfortunately, support for employers’ concerns is found in Table 4. For graduating 
students there are statistically significant, but weak, correlations (.151 and .193 
respectively) between analytical and communication skills and cumulative grade point 
average (GPA). Smaller yet statistically significant correlations of .055 and .082 are 
found between basic numeracy and basic computer skills and GPA. Correlations 
between personal and comparative skills (-.044 and -.045 respectively) are small and 
negative but statistically significant. In essence, there is at best a weak relationship 
between some skills and cumulative GPA of graduating students: marks may measure 
subject knowledge, but not skills. 

Unfortunately, once again, we are left with a quandary. If we assume that university 
graduates must possess the generic skills analysed here, then we must also conclude that 
the skill measures employed in the study are invalid. If, on the other hand, we accept the 
views of critics that university curricula do not necessarily instill skills and that grades do 
not also measure skills, the findings help validate the skills measures employed in the 
study. As some other researchers (Franklin, 1995; Money, 1996) have also reported 



lower than .4 suppressed, the analysis yielded factors similar to those of basic numeracy, 
computer skills, personal skills, comparative skills, and job procuring skills. The other factors 
combined, in no systematic fashion, variables that comprised the analytical and communication 
skill categories (see Spector (1992:54) for a rationale). On the basis of these findings it was 
deemed appropriate to retain the original indices. 
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weak relationships between measures of various skills and grades, it is tempting to side 
with employers. 

While the link between skills and GPA may be weak, in Table 4, the correlation between 
cumulative GPA and OAC marks is a statistically significant .636. This relationship is 
comparable to the findings of studies of graduating students in the United States (Astin, 
1993:188). In essence, OAC marks maintain a reasonably strong relationship with 
academic achievement over the students’ university careers. Indeed, the correlation of 
OACs with cumulative GPAs of graduating students is higher than the correlation with 
first year grade point averages. This observation may be explained by the probability 
that students with low GPAs are less likely than others to finish their education. 



Control Variables 

In the analysis of net gains in various skills, it is important to control for possible 
confounding variables such as gender, racial origin, and language spoken in the home 
while the student was growing up. That each may have implications for particular skills 
is evident from Tables 5, 6, and 7 that include unstandardized skills scores and grades for 
entering and graduating students combined. 

Unfortunately, space constraints prevent a detailed examination of data in the table. At a 
very general level, however, it is obvious that considerable differences in skills are based 
on gender, racial origin, and home language. For example, from Table 5 it is seen that 
males score highest on basic numeracy, and computer skills. Females do better than 
males on analytic, personal, and organizational skills. There are no statistically 
significant differences for comparative skills. 

If ethno-racial origin as summarized in Table 6 is examined, Black students score highest 
on analytic and communication skills. Students of European origin have the highest 
scores on personal and organizing skills while South Asian origin and other students do 
best on comparative skills and South Asian origin students score highest on computer 
skills. The highest scores on basic numeracy skills are reported by students of Chinese 
origin. It must be stressed that absolute differences among some scores for different 
ethno-racial groups are very low. 

One cause for alarm is the low rating for students of Chinese origin on analytic, 
communication, and comparative skills. As the majority of these students did not speak 
English in their homes, these figures likely illustrate the impact of language on various 
skills areas. In a relatively language neutral area, like basic numeracy, students of 
Chinese origin score high. 

The importance of language is further illustrated when language spoken in the home is 
examined. Table 7 shows that students who spoke Italian while growing up score highest 
in analytic, personal, organizing, comparative, and computer skills. Students who spoke 
English do best in communication skills. Finally, students who grew up speaking 
Chinese have the highest scores in basic numeracy. 



Using Surveys to Measure 'Value Added’ in Skills 1 0 



ERiC 



24 








CD 


CO 


CN 


3 


CN 


CT 


tj- 


in 


CD 




* 


D~ 


h- 


CO 


1 — 




C- 


00 


CO 




* 

< 


in 


CN 

CN 




in 


CN 




in 


3 






CL 






















o 


0 




CO 


CT 


CO 




0 




3 




O 

< 


0 


S 




CD 


O 


O 


CT 


CN 




00 

r- 


CT 


CD 


r-' 

r- 


O 


r- 


D- 

r- 


CT 

CN 


CD 




o 






















* 


co 


CO 


s 


CD 


r- 


CD 


Tf 


O 


CD 




* 


CO 


O 


CO 




CT 




CN 


CT 




k— 

a) 


CO 


CO 

CN 




CO 


CN 




CO 


in 

CO 






d 






















Q. 






















E 






















o 






















O 






















* 








0 




in 


00 


CN 


CD 




* 

* 

>N 

o 




CO 

CN 


D-- 


CD 


CO 

CN 


in 




3 

CO 


CD 




2 






















0) 






















E 






















13 






















z 






















0) 


CT 


CO 


CO 


in 


XJ- 


CT 






CD 


k. 

0) 

T3 


> 

2 


CO 

CO 


s 

CN 


CT 


■n- 

CO 


CO 

CN 


CT 


N- 

CO 


d- 

m 

CO 


CT 


C 

a> 

O 


CD 

CL 

E 




















>s 


o 




















jQ 


O 




















to 






















0) 






















T3 


* 


m 


D- 


O 


CO 


CO 


3 


O' 


0 


CO 


2 


* 


in 


CN 


00 


CN 


CO 




CD 


00 


o 


o> 

c 


CO 


CO 

CN 




CO 


CN 




CO 


in 

CO 




■o 


N 




















c 

cd 


'c 

CD 




















(0 


CT 




















2 


O 




















o 






















o 






















<0 


* 


CO 


r- 


CN 


m 


NT 


CD 


CO 




54 


w 


« 


CN 


0 


in 


p 




in 




CN 


£ 


* 

id 

c 




CO 

CN 






CN 




^r 


in 

CO 




w 


o 






















2 




















U> 


aj 




















0) 


CL 




















n 






















<0 


* 


00 


CT 


CO 


CT 




CT 


m 


CO 


in 




* 

* 

c 

o 


0 


CO 

CN 


CD 


CT 

CO 


CN 

CN 


CD 


0 

Tf- 


3 

CO 


cq 




ft 






















o 

'c 






















ZJ 






















E 






















E 






















o 






















O 






















*0 


in 

0 




3 


O 

O 


CO 

CO 


3 


s 


0 

m 


3 




'■£> 


'N- 


CO 

CN 






CN 






in 

CO 






cd 






















c 






















< 






c 






c 






c 










0 






0 






0 










CD 






CD 






CO 










*> 






■> 






’> 










a) 






0) 






0) 






c 




Q 


c 




a 


c 




a 






CO 






<D 






CD 




B 






a) 






a) 




B 


0) 








2 


Z 


00 


2 


Z 


00 


2 


z 


CO 




aj 


a) 




















TJ 

C 

a) 


aj 

E 

0) 






a> 

CD 






"ro 

0 








O 


ll 






2 






H 







CT) 

C/) 



LU 



g> 

to 



CD 

CN 



% 



Pu 

O 



cj 

H 

cn 



m 

eg 



vO 

cn 



CD 

*c 

O 

n 

0 

03 

Q£ 

1 

o 



(A 

© 

■o 

2 

o 

■o 

c 

03 

(0 

£ 

o 

o 

CO 

0) 



CO 

<0 

© 

03 





r- 


00 


T— 


T— 


00 


CO 


5 


00 


CD 


to 




00 


CO 


r- 


CN 




m 


03 


* 




CD 


CO 


CO 


CD 




o; 


3 


in 


in 


CM 




00 


oo 


CO 


I s - 


in 


CO 


« 

< 


lO 




r— 


iri 


*"* 


T— 


to 




in 




T— 


in 


CM 

CN 


T— 


iri 


3 


T— 


Q- 

o 








































s 


to 


CO 


in 


to 


CO 


CN 


CD 


CO 


T— 


3 


to 


CN 


CN 


CO 




CO 


3 


* 


^r 


to 


o- 


to 


O' 


O 


O 




CO 


CM 


o 


CO 


in 


CD 


o 


* 

o 

o 


to 




<6 


r- 




<6 


CD 


CO 


to 


r- 


CO 


r- 


to 


CD 


to 


r- 


CD 

CN 


to 


r- 






f- 






r- 






r- 






r- 




I s - 


* 


to 


s 


o 


r- 


r- 


s 


tn 


r- 


m 


m 


CO 


co 




CD 


to 




CO 


to 


* 


to 


o 


o- 


CD 


N - 


3 


CD 


O’ 


CN 


<J> 




T- 


CD 




CD 


CD 


© 


CO 


CM 


T— 


CO 




T— 


CO 




CO 


N- 




CO 


co 

CN 




CO 


3 




3 






































CL 






































E 






































o 






































O 






































* 


in 


o 


CM 


s 


to 


o 


If) 




to 


m 


r- 


CD 


CO 


m 


CD 


00 


x— 


to 


* 


^r 


o 


r- 


CD 


to 


to 


NT 


TT 


in 


CN 


in 


N- 


3 

CN 


to 




T— 


to 


S' 

2 




CM 




N- 






N- 


CO 






N- 








N* 


in 

CO 




a> 






































E 






































3 






































z 






































* 


lO 


r- 


r- 






to 


r- 


00 


CD 


CN 




r- 


to 


O 


m 


T , 


r— 


to 


* 

* 

a) 

> 


^ 3 - 

CO 


o 

CM 


CD 


to 

CO 


o 

CN 


CD 


CD 

CM 


3 


00 


N" 

CO 


CO 

N- 


CD 


TT 

CO 


to 

CO 

CN 


CD 


CO 


3 

CO 


CD 








































2 






































03 






































Q. 






































E 






































o 






































O 






































* 




s 


00 


CO 


CD 


CD 


rr 


CN 


to 


3 


CO 


CO 




oo 


CN 




CD 


CO 


« 


n- 


00 


CN 


CD 


00 


r— 


3 


r- 


CN 


oq 


m 


m 


°0 




CN 


00 


O) 

c 


CO 


CN 




CO 






CO 




CO 


M- 




CO 


CO 

CN 




CO 


in 

CO 




n 






































c 






































to 






































e> 






































O 






































♦ 




s 




00 


r- 


r- 


CD 


to 


00 


m 


m 


CN 


CN 


CN 




1 — 


3 


54 


5 


CN 


to 


T— 


CD 


U") 


I s - 


CO 


m 




CN 


m 


CN 


CO 


in 




03 

C 


M- 


CN 










co 


CO 




N- 


N- 






CO 

CN 






3 




o 






































2 






































aj 






































a 






































* 


CD 


CO 


00 


s 


I s - 


64 


CO 


3 ; 


CO 


CD 


I- 


m 


N- 




o 


in 


CN 


m 


♦ 

* 




o 


in 


CD 


NT 


NT 


r— 


CD 


CN 


to 




3 

CN 


q 


o 


r— 


to 


c 

o 


V 


CN 




Tf’ 


T ~ 




CO 


CO 




CO 












m 

CO 










































o 

'c 






































3 






































E 






































E 






































o 






































O 






































* 


to 


CO 


CD 




CD 


C 0 


r- 


N" 


r- 


O 


m 


64 


TT 


O 


CO 


CO 




64 


♦ 

* 


T— 


o 


in 


CD 


CD 


CO 


co 


3 


cq 


o 


CN 




m 


in 


o 


CM 


O 




CN 




CO 






CO 












CO 

CN 




O' 


in 

CO 




© 






































C 






































< 






c 






c 






c 






c 






‘c 






c 








o 






o 






o 






o 






o 






o 




















•-P 


























CO 






m 






(0 






CO 












CO 








‘> 






‘> 






’> 






’> 






’> 






'> 








0) 






0) 






0) 






<13 






03 






0) 




c 




Q 


c 




D 


c 




G 


c 




o 


C 




O 


c 




G 




03 






(0 






CO 






m 






03 






03 








a> 




T3 


0) 




T3 


0) 




T3 


0) 




T3 


a > 




TJ 


0 




s 




2 


z 


W 


5 


z 


0) 


5 


z 


W 


5 


Z 


W 


5 


z 


W 


5 


Z 


CO 




























c 


























0) 












03 


























t/) 












a) 












Origin 


Black 






South 


Asian 




<D 

C 

!£ 

O 






Other 






Q. 

2 

3 

LU 






Total 







CD 

<75 



O) 

'</> 



GO 

cv 



CM 



ERIC 



d> 

o> 

re 

3 

m 

c 

re 

_i 

o> 

E 

o 

X 

>» 

0} 

d> 

■o 

E 

O 

T3 

C 

re 

w 

£ 

o 

o 

<0 

</) 



a? 

A 





CNI 


in 


CO 


r*- 


00 


CD 


CM 


CM 


oo 


CM 


CD 


CD 




<<* 


CD 


J 


oo 


cn 


CO 


tt 


CO 


CD 


N* 


00 


4- 


m 


CD 


TT 


r*. 


00 


CO 


< 


in 


M- 

CM 


T- 


in 


CM 


T- 


in 


4— 


T- 


in 


in 


T- 


in 


M- 

CO 


T- 


CL 




























O 


CD 


in 




o 


00 


co 


r*- 


CD 


m 


CO 


CO 


•*— 


CD 


CM 


in 


* 


00 


o 


CD 


o 


o 


M; 


o 


m 


co 


00 


m 


h* 


00 


CM 


CD 


a 




CM 


CO 


a> 


CM 


CD 


r*-’ 




CD 


r'* 




cb 


i'* 


CD 

CM 


cb 


< 


r*. 




r** 






r*. 






r*. 






r*. 




o 
































k_ 


m 


CO 


in 


o 


r*- 


I s *. 




r*- 


r*- 






CD 


tT 


OO 


CD 


CD 


tj- 


CM 


cn 




CO 


CD 


in 


oo 


o 


tT 


r*- 


CD 


tJ; 


i - 


CD 


3 

CL 


CO 


in 

CM 




CO 


CM 




CO 






CO 


in 




cb 


m 

CO 




E 
































o 
































O 
































♦ 




I s *- 


cn 


00 


M" 


CO 


CD 


CD 




00 




r*- 


00 




CD 


* 




^r 


CD 


CD 


CO 




M; 


00 


CD 


m 


I s *. 


in 


M; 


M- 


CD 


V 

o 




in 

CM 






CM 












in 




M 1 


m 

CO 




2 
































<D 
































E 
































3 
































z 
































* 


CM 


CO 


in 




I s *. 


00 


(D 


o 


O 


CD 


CO 


CD 




CD 


CD 


* 




CD 


CD 


CD 


CO 


00 


CD 


cn 


CD 


M; 


oo 


CD 




r*- 


CD 


a) 

> 


CO 


in 

CM 




cvi 


CM 




CO 


T— 




CO 


in 




cb 


m 

CO 




’•+-» 
































2 
































re 
































CL 
































E 
































o 
































O 
































* 


cn 


cn 


CO 


cn 


CO 


in 


CD 


cn 


CM 


o 


00 


CO 




CD 


CO 


* 

« 




m 


00 


o 


CO 


r*- 


CD 


oo 


00 


CO 


I s *. 


00 


xr 


in 


00 


cn 

c 


CO 


m 

CM 




CO 


CM 




CO 






CO 


in 




cb 


in 

CO 




N 
































C 
































re 
































cn 
































6 
































* 




CD 






1^. 


00 


00 


r*- 


CD 




oo 




r*. 




CO 


* 


CM 


CO 


in 


r*- 


CM 


in 


CM 


00 


in 


y— 


CD 


in 


T~ 


CM 


in 


TO 




in 

CM 




CO 


CM 




^r 






M 1 


in 




^r 


in 

CO 




C 






























o 
































(0 
































cd 
































CL 
































* 


in 


o 


CD 


o 


CO 




tj- 




CO 


OO 


CM 


00 


»n 


CM 


CD 


♦ 




M" 


in 


CO 


CO 


r*. 


V— 


00 


CD 


00 


00 


CD 


o 




CD 


c 

o 




in 

CM 




CO 


CM 




M- 


T “ 




CO 


in 




M- 


in 

CO 




































o 

'c 
































3 
































E 
































E 
































o 
































o 
































♦ 


CM 


CM 


CD 


CD 


CD 


in 


CD 


CD 


00 


CO 


CO 


in 


M - 


o 








in 


in 


CM 


CO 


CD 




00 


in 


CD 


r** 


CD 


O 


in 


CD 


O 

s; 


'*}■ 


in 

CM 




CO 


CM 




''fr 






CO 


in 




M- 


in 

CO 




TO 
































C 
































< 






c 






c 






c 






c 






c 








o 






o 






o 






o 






o 








































re 






re 






re 






re 






re 








*> 






*> 






*> 






> 






’> 








d> 






(D 






CD 






CD 






d) 




c 




Q 


c 




Q 


C 




Q 


c 




Q 


C 




a 




re 






re 






re 






re 






re 








cd 

2 


Z 


■d 

CO 


d) 


z 


2 

if) 


<D 


Z 


2 

if) 


CD 

2 


Z 


2 

if) 


CD 


Z 


*D 

4-* 

if) 


cd 
































cn 








<1) 
























re 


x: 






to 
























3 

cn 


« 






d) 

c 






c 

re 






L_ 

CD 






"re 






c 


CD 


















A 












(0 


c 






JC 






re 






<♦— > 






o 






_l 


LU 






O 












o 






1- 







o 

CO 



o 

o 



LU 



o 

LU 



g> 



LO 

o 



LU 



D> 

C/> 



GO 

CQ 



o 

ERIC 



Assessing Value Added 

As noted earlier, in order to determine value added, it is necessary to compare the scores 
on various skill indices of graduating to entering students. While standardized scores are 
not necessary to achieve this objective, there were two reasons for calculating them in the 
current undertaking. (For most practical purposes standardized scores (or z-scores) have 
a mean of 0 and a standard deviation of 1 .) First, in order to compare changes at York to 
changes reported in the literature, and to make changes understandable to the average 
reader, it is necessary to specify differences between graduating and entering students in 
terms of percentile point differences. Second, standardized rather than actual scores will 
be reported in order to protect the confidentiality of the York data. In essence, while 
readers will know the amount of change that can be attributed to the university 
experience, they will not know the absolute value of entering and graduating scores 
(however, such data are available to the York community). From a value added 
perspective the former information is more important than the latter: the concern is not 
with the level of skills of graduating students but with how much the university 
experience has contributed to the development of skills. 

Percentile point difference in the skills of graduating as compared to entering students in 
the Schulich School of Business (Business), Arts, Fine Arts, and Pure and Applied 
Science (Science), after adjusting for OAC marks, gender, ethno-racial origin, and home 
language, are presented in Table 8. The specific skills analysed are listed in column one. 
Column two, ‘Entering Students’, lists standardized scores for each skill by faculty for 
students entering the university. Similar measures for graduating students are found in 
column three, ‘Graduating Students’. For columns two and three the number of cases on 
which the scores are calculated are found in parenthesis. In column four, differences in 
scores for entering and graduating students are presented as percentile point differences. 

To begin with analytic skills, the highest scores for entering students are found among 
students entering Fine Arts (-. 1 8) and Arts (-.34). Among graduating students, those 
from Fine Arts (.40) and Arts (.22) have the highest scores. In terms of the difference 
between the scores of entering and graduating students, however, the greatest 
improvement, or value added, is observed for Business students (26 percentile points). 

With communication skills, students entering Fine Arts have the highest scores (-.23) and 
Science students the lowest (-.69). Among graduating students those from Business have 
the highest scores (.44). Moreover, the value added in communication skills (32 
percentile points) is higher once again in Business than in any other faculty. 

Students entering Fine Arts report the highest scores on personal skills (.07). Among 
graduates the highest scores are for Arts students (.16). In terms of value added, 
however, Business students show the greatest improvement (21 percentile points). 

Among entering students, those going into Arts have the highest organizing scores (-.30); 
however, among graduates, those from Business score highest (.44). Business students 
also show the greatest value added (29 percentile points). 

When it comes to comparative skills, entering Business students do better than others 
(.05). Upon graduation, however, Arts students score highest (.16). Arts students also 
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Table 8: Skills' Z-Scores by Faculty for Entering and Graduating Students 
(Controls on OAC marks, gender, ethno-racial origin, and home language) 





Entering Students 


Graduating Students 


Percentile Difference 


Analytic Skills 








Business*** 


-0.55 (95) 


0.13 (62) 


26 


Arts*** 


-0.34 (760) 


0.22 (1229) 


22 


Fine Arts*** 


-0.18 (158) 


0.4 (138) 


23 


Science*** 


-0.56 (207) 


-0.29 (122) 


10 


Communication Skills 








Business*** 


-0.38 (92) 


0.44 (61) 


32 


Arts*** 


-0.38 (758) 


0.23 (1295) 


24 


Fine Arts*** 


-0.23 (160) 


0.34 (137) 


22 


Science*** 


-0.69 (211) 


-0.13 (121) 


20 


Personal Skills 








Business*** 


-0.56 (90) 


0.02 (62) 


21 


Arts*** 


-0.17 (755) 


0.16 (1300) 


13 


Fine Arts 


0.07 (151) 


0.09 (138) 


0 


Science 


-0.28 (199) 


-0.23 (123) 


2 


Organizing Skills 








Business*** 


-0.32 (92) 


0.44 (62) 


29 


Arts*** 


-0.3 (758) 


0.2 (1306) 


20 


Fine Arts*** 


-0.43 (158) 


0.08 (138) 


17 


Science*** 


-0.52 206) 


0.09 (123) 


20 


Comparative Skills 








Business 


0.05 (94) 


0.08 (62) 


0 


Arts*** 


-0.1 (765) 


0.16 (1304) 


10 


Fine Arts 


-0.13 (161) 


-0.21 (138) 


-3 


Science 


-0.5 (215) 


-0.47 (123) 


1 


Basic Numeracy 








Business 


0.41 (96) 


0.53 (63) 


4 


Arts*** 


-0.28 (753) 


0.05 (1290) 


11 


Fine Arts*** 


-0.65 (160) 


0.08 (137) 


24 


Science* 


0.45 (215) 


0.56 (122) 


4 


Computer Skills 
Business*** 


0.14 (90) 


1.05 (62) 


30 


Arts*** 


-0.26 (755) 


0.03 (1292) 


10 


Fine Arts*** 


-0.38 (155) 


0.08 (136) 


15 


Science*** 


0.06 (206) 


0.66 (124) 


25 



Significance of F for differences between entering and graduating students: * LE .05; **LE .01; ***LE .001. 
Number of cases in parenthesis. 
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display the greatest value added (10 percentile points). It is worth noting that for Fine 
Arts students there is a net loss in terms of comparative skills between first and final year 
(-3 percentile points). As it is unlikely that a Fine Arts curriculum relates to comparative 
skills as measured here, this is not surprising. 

With scores of .45, students entering Science have the highest basic numeracy scores 
among first year students. Science students also have the highest scores among 
graduating students (.56). In terms of value added, however, the greatest improvement is 
found for Fine Arts graduates (24 percentile points). It should be noted, however, that 
the final Fine Arts score is only .08. 

Among entering students, those in Business have the highest scores on basic computer 
literacy (.14). This score increases to 1.05 for Business graduates and represents a gain 
of 30 percentile points, the highest increase of students in any faculty. 

There are two ways in which the data in Table 8 can be examined. First, it can arbitrarily 
be decided that for graduating students a score of, say, . 10 is too low in an absolute sense 
for any skills under discussion. If this yardstick is accepted, the scores of graduating 
Science students for analytic (-.29), communication (-.13), personal (-.23), organizing 
(.09), and comparative skills (-.47) are unacceptable. Similarly, the personal (.02) and 
comparative (.08) scores of Business graduates are problematic. The problem areas for 
Fine Arts graduates are personal skills (.09), organizing skills (.08), comparative skills (- 
.21), basic numeracy skills (.08), and computer skills (.08). Arts students are low in the 
areas of basic numeracy (.05) and computer skills (.03). 

The second way in which the data can be examined is in terms of value added. If this 
tack is followed, it can be seen that the greatest gains in analytic, communication, 
personal, organizing, and computer skills are made by students in Business. The value 
added in comparative skills is greatest for Arts graduates and Fine Arts students show the 
greatest increases in basic numeracy. 

Overall, the information in Table 8 indicates first, that some skill levels of graduating 
students in certain faculties are too low. Second, faculties differ in the amount of value 
added to particular generic skills. 

In order to obtain an overview of value added in the skills under discussion, we can 
simply take the average over the four faculties. When this is done, in descending order 
average gains are: communication skills (25 percentile points), organizing skills (22 
percentile points), analytic and computer skills (20 percentile points each), basic 
numeracy skills (1 1 percentile points), personal skills (9 percentile points), and 
comparative skills (2 percentile points). 



The Effects of Maturation 

As noted earlier, it was not possible to have an external same age control group of 
individuals who did not go to university. As a result, there is a possibility that the skills 
gains noted in the previous section could be totally or in part a result of maturation. In 
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order to test for this possibility, using the results of surveys carried out for departmental 
reviews, the skill scores of first year students in Atkinson College were compared to 
those of students in other undergraduate faculties. The average age of first year Atkinson 
students was 32; in other faculties it was 21. The mean number of courses registered in or 
completed for Atkinson students was .8; for other students, 4.4. 

The analysis was restricted to first year students who had no prior post-secondary 
education at either the college or university level and skill scores were adjusted for the 
effects of gender, ethno-racial origin, and language spoken in the home while growing 
up. Adjustments were not made for OAC marks because admission to Atkinson is not 
necessarily based on high school performance. 

The results of analyses of covariance indicate that with one exception there were no 
statistically significant differences between first year students in Atkinson and in other 
faculties or colleges on the measures of skill used in this study. The exception was 
organizing skills for which the mean score of Atkinson students was 1 1 percentile points 
higher than scores of other students. Findings such as these give support to the notion 
that differences between entering and graduating students analysed earlier are not simply 
the result of maturation. 

This said, a note of caution must be sounded. Surveys for program reviews are carried 
out between January and March. As a result, it is possible (but not probable) that the 
skills of students in colleges or faculties other than Atkinson could have differentially 
increased since the beginning of the year. 




An Overall Assessment 



How can we evaluate the differences in generic skill scores of students entering and 
leaving York University? Fortunately, in their tome, How College Affects Students , 
Pascarella and Terenzini (1991:558) have summarized findings from a number of studies 
on freshman to senior gains in a number of domains similar to those analysed here (see 
Table 9). 

The first column of the table lists the skills under consideration. Column two lists the 
‘effect size’ that can be viewed as the difference between the mean score for graduating 
and entering students divided by the entering student standard deviation. (This value is 
equivalent to the unadjusted mean in the analysis of covariance conducted for this report. 
Elsewhere (Grayson, 1996) it has been shown that for the Faculty of Arts at York, 
differences between unadjusted and adjusted means for the skills scores are minor.) 
Column three contains information of effect size translated into area under the normal 
curve. 

There are two important observations that can be made on the basis of the data in Table 
9. First, overall gains between first and final year are modest ranging from a low of 10 
percentile points for quantitative skills to a high of 38 percentile points for ability to deal 
with conceptual complexity. Second, while different measurement techniques preclude 
direct comparisons, there is a enough similarity between differences observed at York 
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and those seen in studies conducted elsewhere to inspire confidence in measures used for 
the York study. 

For example, Pascarella and Terenzini report gains of 21 percentile points for general 
verbal skills, 22 points in oral communication skills, and 19 percentile points in written 
communication skills (the average for these three is 21 percentile points). At York, 
average gains in communication skills, that included some general verbal as well as 
written and oral components, were 24 percentile points. The same is true of quantitative 
skills: Pascarella and Terenzini report gains of 10 percentile points and the average gain 
in basic numeracy at York was 1 1 percentile points. 

It must be stressed that the studies summarized by Pascarella and Terenzini used many 
different measures of the skills under consideration (some involved the use of 
standardized tests of various skills). As a result, it would be wrong to place too much 
emphasis on specific comparisons between their and the York findings. What is 
important is the fact that on the two gains on which it was possible to comment, increases 
observed at York were similar in size to those measured elsewhere. 



Broader Implications 

Students graduating from four faculties at York University report higher skills than 
entering students. Unfortunately, the degree to which value added in generic skills at 
York is comparable to that in other Canadian universities is contingent upon the latter 
opening comparable avenues of inquiry and publicizing the results. 

In the event that similar research were initiated, what might be expected? Research 
carried out in the United States can be of assistance in answering this question. As 
Pascarella and Terenzini (1991 :589), after their review of approximately 3,000 U.S. 
studies related to university outcomes, argue: 

There are clear and unmistakable differences among postsecondary institutions in 
a wide variety of areas, including size and complexity, control, mission, financial 
and educational resources, the scholarly productivity of faculty, reputation and 
prestige, and the characteristics of the students enrolled. At the same time, 
however, American colleges and universities also resemble one another in a 
number of important respects. It may be that despite their structural and 
organization differences, their similarities in curricular content, structures, and 
sequencing; instructional practices; overall educational goals; faculty values; out- 
of-class experiences; and other areas do in fact produce essentially similar effects 
on students although the 'start 'and 'end' points may be very different across 
institutions. 

Given that U.S. universities are far more diverse than those in Canada, it is likely that in 
this country there is even a smaller difference in outcomes such as those studied here 
than south of the border. In other words, while institutions like York, Memorial, 

Queen’s, Ryerson Polytechnic University, the University of Toronto, Western, the 
University of Manitoba, the University of Regina, and the University of Victoria may 

Using Surveys to Measure 'Value Added’ in Skills 14 



BEST COPY 



37 



vary in things like resources and the entering average of students, based on findings from 
U.S. research, it is unlikely that great differences exist in the value added to skills similar 
to those studied at York. 



Conclusion 

The objective of this study was to assess the value added to generic skills by the 
university experience. In the best of all possible worlds such a study would have 
measured the skill levels of entering students using both self-reports and standardized 
tests and then, after the students had completed their education, skills measured with the 
same instruments could be compared to those at entry. Provided that increases in skills 
were not also observed in a same age control group that did not attend university 
differences between entry and exiting skill levels could be attributed to the university 
experience. 

Unfortunately, this type of research is both costly and time consuming and unlikely to be 
carried out in Canadian universities; however, the type of research described in this 
report, in which, after adjusting for potentially confounding influences, self-reports of 
skill levels of entering students were compared to those of graduating students, is a cost 
effective and timely alternative to collecting information in longitudinal studies. The 
justification for taking this tack can be found in previous research indicating acceptable 
levels of agreement between self-assessed and other measures of skills and the similarity 
in results of cross-sectional and longitudinal research carried out in related areas. 
Although information collected via the process described in this study should not be used 
in decisions regarding individuals, it does have utility in assessing change in groups. 

Information collected in this fashion can have several practical applications. First, once 
benchmark have been established, within institutions, individual faculties can determine 
whether or not relative to other faculties their students emerge from their courses of 
studies with acceptable skill levels. Second, faculties that make deliberate attempts to 
enhance certain skills can see if there are differences in skill levels between entering and 
graduating students. In either the first or second cases, if skill levels are lower than 
desired, steps can be taken to remedy the problem. Third, if similar information were 
available from other institutions, universities could compare the value added by their 
institution to that of other universities. Once again, if comparisons were unfavourable, 
changes could be introduced to deal with deficiencies. 

Consistent with the foregoing possibilities, the most important finding of the current 
study was that, in keeping with previous research, students graduate from York with 
better developed skills than students who enter the institution. There are, however, 
important differences between and among faculties in terms of the value added to 
particular skills and in some faculties the skill scores of graduating students may be too 
low. Moreover, some evidence suggests that the differences observed between entering 
and graduating students are not the result of maturation. Furthermore, the magnitude of 
two average gains in skills are comparable to gains recorded in the United States. Given 
the results of research conducted south of the border, similarity in results such as these is 
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to be expected. Unfortunately, we will not be able to compare the value added at York to 
that of other Canadian universities until the latter conduct the required research. 
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